General Disclaimer 


One or more of the Following Statements may affect this Document 


• This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 


• This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 


• This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 


• This document is paginated as submitted by the original source. 


• Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 


Produced by the NASA Center for Aerospace Information (CASI) 



NASA 

Technical Memorandum 78092 


The Effects of Quantization 
on Signal Processing 

(NASA-TH-78092) THE EFFECTS CF QOANTIZATICF N78-2 

ON SIGNAL EBCCESSING (NASA) 25 p fiC A02/nF 
A01 CSCI 17E 

Oncla 
G3/32 12573 


H. Montgomery and E. Schell 


FEBRUARY 1978 


National Aeronautics and 
Space Administration 

Qoddard Space Right Center 

Greenbelt. Maryland 20771 





j 


C-^ 



THE EFFECTS OF QUANTIZATION ON 
SIGNAL PROCESSING 


H. Montgomery 
E. Schell 


February 1978 


GODDARD SPACE FLIGHT CENTER 
Greenbelt, Maryland 


THE EFFECTS OF QUANTIZATION ON 
SIGNAL PROCESSING 

H. Montgomery 
E. Schell 

ABSTRACT 

Typically an analog signal from a space system is sampled, quantized by 
Analog-to-Digital (A/D) conversion, merged into a bit stream, communicated 
to a ground station, received by the ground station, and processed by the 
ground station to extract useful information for dissemination to the users. 
The cost of each of these steps is reduced as the number of quantization steps 
is reduced in the A/D converter. The number of quantization steps should 
be as small as possible without losing the required information content. This 
report deals specifically with the accuracy of averages as a function of the 
number of quantized samples used to compute the averages with the noise 
on the analog signal as a parameter. For example, the success of the Visible 
Infrared Spin Scan Radiometer (VISSR) Atmospheric Sounder (VAS) 
Demonstration depends upon temporally averaging multiple samples in an 
effort to reduce noise to a sufficiently low level such that temperature pro- 
file sounding is made possible. A tutorial description of this process is pre- 
sented which demonstrates the viability and limitation of noise reduction by 
averaging quantized samples of an analog signal. 
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= The standard deviation of the set of averages A about the mean M = T. 


On = 




= The nns error incurred by quantizing a sample of a clean (noise free) signal. 


o 
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THE EFFECTS OF QUANTIZATION ON SIGNAL PROCESSING 

Averages computed from N samples of an analog signal have a standard deviation 
about the true signal given by (see Appendix A) 


where 



( 1 ) 


o, = The standard deviation of the noise per sample of an analog signal. 

N = The number of samples of an analog signal which are used to compute the average. 

= The standard deviation of the set of averages. Each average is determined from N 
samples of an analog signal. 

Equation 1 does not include the effects of quantization errors. Quantization of a perfectly 
clean sample (noise free) of an analog signal introduces a root mean square (rms) quantiza- 
tion noise of (see Appendix B): 

Prms quantization nois^ I 

of a sample of a clean I = — -:z:r quantization steps (2) 

|_signal J vl2 

Both noise on the analog signal and quantization errors degrade the accuracy of the 
average of N samples. Figure 1 presents the combined effect on the accuracy of the average. 
This figure gives Oqi^i (the standard deviation of the averages) as a function of N (the num- 
ber of quantized samples of an analog signal used to compute the average) for various values 
of Oj (the standard deviation of the noise per sample of the analog signal). The method of 
computation is described in Appendix C. Table 1 presents the asymptotic values of 
as a function of Oj for very large values of N(N -► «»). See Appendix D. It 
should be noted that the curve for Oj = .2 in Figure 1 is asymptotic to - .1023 as is 
indicated in Table 1. 


1 



Table 1 


Asymptotic Values OnN as a Function of o, 
QN(n.oo) 


— ■ 


0 

0.288675134= l/y/Tl 

0.100000000 

0.192150011 

0.200000000 

0.102308167 

0.300000000 

0.038089263 

0.400000000 

0.009565700 

0.500000000 

0.001618753 

0.600000000 

0.000184583 

0.700000000 

0.000014182 

0.800000000 

0.000000734 

0.900000000 

0.000000026 

1.000000000 

0.000000001 

1.100000000 

0.000000000 

0,> 1.1 

0 


The set of averages where each average is determined from N quantized samples has a 
standard deviation about the true analog signal of Ogjig. If the standard deviation of the 
noise of the analog signal is o^, then 


and 






— OnN <—=foT0<o,< l.N> 12 
v/n /iT 


^QN 


— = for o. > 

y/N * 


where o, and Ogfi^ are measured in unit quantization steps. For 0 < < I and 

N < 1 2 no generalization can be made with regard to the value Ogi^. however Figure 1, does 
give specific results for Oj = 0. 1 , 0.2, 0.35 and 0.5. 
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APPENDIX A 


VARIANCE OF THE ANALOG AVERAGE 

Let Sj be the i*** sample of an analog signal whose true value is T, then Sj is given by 


Sj « T + f?, 


•n 


where 

T = The true signal ■ a constant 

* Gaussian noise with zero mean and standard deviation o, 

The average A of N samples is computed from the expression 

N 


A * 



Si 


Substituting Eqn. (A*l) into Eqn. (A-2) the following result is obtained 



i-I 


The mean M of the average A is computed as follows: 

N 

N 

or M = T+-J^^E|Tjj| 

1-1 

Since i7| has mean zero. i.e. 

E(t?j) ' 0. 


(A-2) 


(A-3) 


(A-4) 


(A-5) 


(A-6) 


A-1 


OF POOR 


Fqn. (A-5) becomes 


M =T 


(A-7) 


Next compute the variance, of the average A about the mean M as follows 


V. =F|(A-M)‘| 


(A-8) 


Substituting Hqns. (A-3) and (A-7) into tqn. (A-8) results in 

N N 


or 


i-i j-i 


(A-9) 


By assuming thit and 7 j arc independent for i # j, and utilizing the definition of the 
expectation operator, the following lesult is obtained 




= V«ij 


(A-IO) 


where the Kronecker delta is defined by 


0 for i j 
•j I I for i = j 


(A-ll) 


Substitute ton. (A-IO) mio Fqn. ( A-9) to obtain 

•» N N 


EE 


«.j 


But 


N N 


EEvn 


i-i j-i 


(A-12) 


(A-13) 


A-2 






SubsUtLtion of Eqn. (A-I3) into Eqn. (A-1 2) gives the 


variance of ihe average 


V. = — 


The standard deviation of the average is then given by 


(A- 14) 


(A- 15) 
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APPENDIX B 


SINGLE SAMPLE QUANTIZING ERRORS 

Given an analog signal S (measured in quantization step units) which is to be quantized. 
Without loss of generality, it may be stated that the value of S falls in the integer quantiza- 
tion step Q. In other words, S lies anywhere between Q - 5^ and Q + with equal probabil- 
ity. Given this constraint, the mean value of S, Mq, is given by 

X Q+% q2 Q+V4 

SdS=-- 

, -V4 2 

or 


Mq 


= Q 


(B-1) 


If S falls in the integer quantization step Q, the value of S is approximated by Q and 
the err'' S - Q, for one sample, lies between -'A and +A. 

Av, roximating S by Q over a whole population of S’s, the root mean square (rms) 
error Oq in the estimate of S is given by 





(S- Q)2dS 


> 




1 

Oq = quantization steps 


(B-2) 
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APPENDIX C 


VARIANCE OF THE AVERAGE WITH QUANTIZING 

The quantized signal is computed from the expression 


Qijk “ 


(C-1, 


where 

Int (x) = The largest integer which is less than or equal to x* 

The i^^ sample of an analog signal which has Tj for a true value and gaussian noise rj^y^, 
and is used to compute the k^^ average Aqjj^ is computed from 

^ijk “ ’^ik 


Appendix E shows a method of computing gaussian noise q. 

The average Aqji^ of N quantized samples is computed from the expression 


^Ojk 




i“l 


ijk 


(C-3) 


To determine how well Aqji^ approximates the true value Tj, compute the variance 


Vqnj of Aqjij about Tj as follows: 


k«l 


(C-^' 


The standard deviation <^QNj of ^Qjjc given by 


®QNj “ V'*'QNj 


(C-i 


Figures C-1 through C-4 present Oqj^j for Tj = 0, 0.01 , 0.25 and 0.5 respectively. 
These figures demonstrate that for Oj > 1 the values of Oqnj are independent of the value 
of Tj, and Oqnj is a function of Tj for o, < 1. For Tj = 0 (the middle of the quantization 


step) Figure C-1 indicates that OgNj = 0 for o, < 0. 14. Figure C-2 shows that for Tj = 0.01 
(0.01 units from the middle of the step). Oq^j = 0.01 for < 0.14. Figure C-3 shows that 
for Tj = 0.25. OQNj = 0.25 for o, < 0.07. For Tj = 0.5 (tnie value on the threshold between 
quantization steps 0 and 1). the curves Oqnj versus N are identical for < 0.2. 

It is necessary to obtain the average variance for equally likely values of Tj. Thus 
taking R values of Tj spaced uniformly between 0 and 0.5 results in 

^ON ^ 

The standard deviation Oq^ of the average of N quantized samples of an analog signal 
about the true signal which is uniformly distributed over a quantization step is given by 

‘^QN “ \/^QN 


C-2 





Function of the Number of Quantized Samples Used to Compute 




Figure C-2. Standard Deviation of the Average as a Function of the Number of Quantized Samples Used to Compute 

the Average for a True Signal Value Tj = 0.01 



ox - TRUE VALUE AT QUANTIZATION 



Standard Deviation of the Average as a Function of the Number of Quantized Samples Used to Compute 
the Average for a True Signal Value T = 0.25 



APPENDIX D 


qqn as a function of o, for very large N (N -►«») 

Let the true value T of an analog signal be defined by 

T»Q + x, ViKxKVi (D-1) 

This true signal is corrupted by noise (with zero mean and variance o,) and quantized. This 
appendix will show what the variance of the average is about the true signal for averages 
based upon an infinite number of quantized samples. 

Define a quantizing function CKS) for signal S by 

G(S) = Q + iforQ + i- V4<S<Q + i + ^ 


where 


i = -*» -N, ...,-1,0, 1,...,N, ...,«. 


Then the average, S, of an infinite number of quantized samples is given by 


S = ElG(S)l 
= jf G(S)p(S)dS 


or 


S = Q + F(x) 


(D-2) 


where 


F(x)= lim > i / , , P(S)dS 


= S‘X« 


(D-3) 


D-1 


and 


1 


Let 


P(S) = 




g-<S-T)«/2o,* 


(D-4) 


z = S-T (D-5) 

in Hqn. (D*3) and after rearranging and utilizing Eqn. (D-I), obtain 

N 

F(x)= ^ y I. . P(z)dz (D-6) 

where 

p(z) = 

One notes that F(0) = 0, FC/ 2 ) = V 2 and F(-V4) = -Vi. 

The variance V of the average about the true value T is given by 


V= /*^(S-T)2dx 
J-Vi 


or V= J (F(x)-x)^dx 

(D-8) 

But (F(x) - x)^ is an even function of x since 


(F(x)-x]2 = (F(-x) + x]2 

(D-9) 


Hence 


V= 2 



[F(x)“ x)^ dx 


(D-IO) 


<>0- = v^ <D-1I) 

This is the desired result, i.e., the variance of the quantized signal about the true value 
for an infinite number of samples. 


D-2 
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APPENDIX E 


COMPUTATION OF GAUSSIAN NOISE 


The gaussian distribution p(y) is given by 


P(y )“ — —t 

Oy/^ 


where 


y = The random variable 


a = The mean of the random variable y 


a = The standard deviation of y 


0 = 1 . 


Fix y|, i = 1 to w-t- 1 , such that there is equal probability of .S/w for each interval 


given by 


0 = a < yj < y < yj+ j for i = 1 to w 
The y^’s may be computed numerically from the equation 


•yi+i .5 

p(y)dy = — for i = I to w 
. w 


where 


y, =a-0 


V: for 1 = 1 to w 

' 2 
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Let U be a uniformly distributed random variable between 0 and I. Then random 
noise rj with a gaussian distribution of zero mean and standard deviation of o, may be com- 
puted as follows 


i = Int (Uw) + I 

(f4) 

V = o,Vi sgn (U- .5) 

(E-5) 


where U was computed by utilizing a standard IBM program for uniform random number 


generation. 
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